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EXECUTIVE  SUMMARY 


As  part  of  a  previous  contract,  RWDI  was  retained  by  Alberta  Environment  (AENV)  to: 

a)  produce  a  subset  of  Mesoscale  Model  version  5  (MM5)  output  files  at  12  km  resolution 
covering  the  Province  of  Alberta  based  on  model  outputs  for  the  2002  model  year  provided 
by  Environment  Canada  Prairie  and  Northern  Region  (EC);  and 

b)  develop  a  Multi-Model  Extraction  Utility  (MMEU)  that  can  be  freely  distributed  and  used 
with  the  aforementioned  MM5  subset  to  generate  input  files  for  use  with  a  suite  of  air  quality 
models.  Details  are  available  from  the  following  websites: 
http://environment.gov.ab.ca/info/library/7696.pdf;  and, 

http://environment.gov. ab.ca/info/posting.asp?assetid=7696&searchtype=asset&txtsearch=m 
m5.  AENV  has  since  retained  RWDI  to  perform  the  following  tasks  under  a  separate 
contract: 

Task  1.  Review  /  evaluate  the  2002  Environment  Canada  MM5  outputs  over  the  Alberta  subset 
domain. 

Task  2.  Run  the  MM5  model  for  three  consecutive  years  at  12-km  resolution  covering  the 
MMEU  domain. 

Task  3.  Update  the  MMEU  software  solution  to  enable  it  to  ingest  and  provide  outputs  using  the 
new  MM5  data  set  developed  under  Task  2. 

The  remainder  of  this  document  focuses  strictly  on  the  results  of  Task  1  and  serves  as  a  guidance 
document  on  the  use  and  interpretation  of  the  2002  Environment  Canada  MM5  subset  supplied 
by  Alberta  Environment  with  MMEU  (Version  2006.03.03). 
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1.0  BACKGROUND 


The  purpose  of  this  review  is  not  to  perform  a  comprehensive,  statistical  model  evaluation  of  the 
2002  dataset  produced  by  Environment  Canada.  Rather,  the  purpose  is  to  review  the  overall 
performance  of  the  model  in  a  general  sense  for  a  limited  number  of  key  parameters  and  to 
address  some  issues  related  to  the  data  that  were  forward  to  RWDI  by  AENV.  From  this 
analysis,  conclusions  and  general  "usage  notes"  have  been  formulated. 


Review  of  2002  12  Km  MM5  Model  Results 


! 


2.0  METHODOLOGY 


The  model  evaluation  was  based  on  qualitative  and  quantitative  comparisons  between  paired 
hourly  model  results  and  observations  from  eight  surface  and  two  upper  air  stations. 
Environment  Canada  surface  stations  selected  for  the  analysis  were  chosen  to  provide  geographic 
coverage,  with  a  bias  towards  areas  where  air  quality  modelling  is  likely  to  be  performed. 
Although  the  focus  of  the  evaluation  was  on  the  surface  layer,  a  qualitative  assessment  of  model 
performance  aloft  was  also  performed  using  data  from  two  upper  air  (balloon)  stations.  The 
stations  used  are  listed  in  Table  1  and  shown  geographically  in  relation  to  the  MMEU  domain  in 
Figure  1  (blue  symbols  and  font  represent  surface  stations;  red  represents  upper  air  stations). 

Table  1.  Name  and  coordinates  (Latitude  /  Longitude)  for  meteorological  observing 
stations  used  in  the  evaluation  of  Environment  Canada's  12  km  resolution 
MM5  outputs  for  2002  as  adopted  in  MMEU 


Observation  Station 

Latitude  (N) 

Longitude  (W) 

Calgary  Airport 

51°  6.600' 

114°  1.200' 

Edmonton  Airport 

53°  19.200' 

113°  34.800' 

Edson  Airport 

53°  34.800' 

116°  28.200' 

Ft.  McMurray  Airport 

56°  39.000' 

111°  13.200' 

Grande  Prairie  Airport 

55°  10.800' 

118°  53.400' 

High  Level  Airport 

58°  37.200* 

117°  9.600' 

Lethbridge  Airport 

49°  37.800' 

112°  48.000' 

Slave  Lake 

55°  17.400' 

114°  46.800' 

Stony  Plain  (upper  air) 

53°  33.000' 

114°  6.000' 

Great  Falls  (upper  air) 

47°  27.000' 

111°  22.480' 
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Figure  1.        Geographic  coverage  of  the  MMEU  domain  and  location  of  meteorological 

observing  stations  used  in  the  evaluation  of  Environment  Canada's  12  km  resolution 
MM5  outputs  for  2002  as  adopted  in  MMEU  (blue  =  surface  station;  red  =  upper 
air). 
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The  model  evaluation  was  based  on  four  parameters  that  relate  to  key  meteorological  inputs 
required  for  dispersion  modelling:  dry  bulb  temperature,  wind  speed,  wind  direction,  and  relative 
humidity.  Dry  bulb  temperature,  wind  speed,  and  direction  were  extracted  directly  from  MM5 
results;  relative  humidity  was  calculated  based  on  the  vapour  mixing  ratio  from  the  MM5.  The 
review  process  involved  the  following  tasks: 

1 .  Perform  a  qualitative  review  of  the  MM5  model  options  (nesting,  nudging,  physics  options) 
adopted  by  Environment  Canada. 

2.  Extract  and  tabulate  model  outputs  and  observations  for  eight  surface  observing  stations  and 
two  upper  air  stations. 

3.  Produce  station-by-station  time  series  and  1:1  scatter  plots  from  hourly  results  for  both 
Winter  (January)  and  Summer  (July)  months  for  the  following  (Appendix  A): 

•  Temperature 

•  Wind  Speed 

•  Wind  Direction 

•  Relative  Humidity 

4.  Produce  domain- wide  time  series  and  1 : 1  scatter  plots  from  hourly  results  for  both  Winter 
(January)  and  Summer  (July)  months  for  the  following  (Appendix  B): 

•  Temperature 

•  Wind  Speed 

•  Relative  Humidity 

5.  Calculate  simple  performance  statistics  (Fractional  Bias  and  Normalized  Mean  Square  Error) 
on  the  domain-wide  results  for  the  winter  and  summer  periods,  as  well  as  the  entire  year  (i.e., 
annual  statistics  -  no  graphs  produced)  for  the  following  parameters  (Appendix  B): 

•  Temperature 

•  Wind  Speed 

•  Relative  Humidity 
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The  following  statistics  were  computed: 


C  -C° 
Fractional  Bias  oxFB  -  -= — 


(CN  -C"  f 


(Cs  +C°)/2 

Normalized  Mean  Square  Error  or  NMSE 

Cs  •  C° 

where  Cs  =  model  prediction 

C°  =  station  observations 

6.  Produce  time  series  from  hourly  MM5  results  and  twice  daily  balloon  data  for  both  Winter 
(January)  and  Summer  (July)  months  for  the  following  parameters  at  100m,  500m,  and 
1000m  (Appendix  C): 

•  Temperature 

•  Wind  Speed 

•  Wind  Direction 

•  Relative  Humidity  (MM 5  results  only) 

An  excerpt  from  Appendix  A  is  provided  in  Figure  2  to  assist  the  reader  in  interpreting  the 
results  provided  in  the  various  appendices.  Plots  from  other  parts  of  the  analyses  (domain- wide 
statistics,  etc.)  provided  in  Appendices  B  and  C  follow  a  similar  layout. 
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Figure  2.  Sample  time  series  and  scatter  plots. 
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3.0 


MODEL  RESULTS:  DRY  BULB  TEMPERATURE  -  SURFACE 


The  plots  in  Appendix  A  indicate  that  the  model  overestimated  daily  maximum  temperatures  in 
the  summer  at  all  stations.  This  appears  to  be  due  to  an  over-prediction  of  peak  daytime 
temperature  during  a  well-developed  diurnal  temperature  cycle  typical  of  clear  sky,  anti-cyclonic 
synoptic  conditions.  Overnight  lows  were  simulated  well  at  all  stations. 

The  model  slightly  underestimated  temperature  at  all  sites  in  the  winter.  Under-predictions  are 
not  associated  with  a  diurnal  pattern  but  appear  to  be  associated  with  longer,  larger  scale 
synoptic  patterns  that  last  several  days. 

These  seasonal  discrepancies  in  predicted  temperatures  are  most  likely  the  result  of  modelling 
artefacts  associated  with  the  selection  of  the  Medium  Range  Forecast  (MRF)  planetary  boundary 
layer  (PBL)  scheme  (combined  with  other  options),  and  input  data  quality  issues  from  the  2.5 
degree  resolution  NCAR/NCEP  Reanalysis  Project  (NNRP  or  Rl)  data  used  by  Environment 
Canada. 

Using  coarse  resolution  NNRP  data  as  the  primary  input  to  MM5,  without  the  addition  of 
nudging  (FDDA),  has  been  demonstrated  to  result  in  these  types  of  temperature  biases  in  MM5 
results  (Salathe,  2007).  In  addition,  as  other  MM5  modellers  have  noted,  the  MRF  scheme 
consistently  overestimates  the  height  of  the  PBL,  especially  during  the  summer  months. 
Experiments  (ATMET  Report,  2003)  have  shown  this  tendency  to  overestimate  the  depth  of  the 
boundary  layer  especially  during  the  daytime  hours.  Overestimates  manifest  themselves  in 
numerous  ways,  including  too  little  moisture  in  the  PBL  and  an  overestimate  of  daytime 
temperatures  in  summer.  Further,  the  Shallow  Convection  scheme  was  not  used  in  either  the  12 
km  or  parent  36  km  resolution  domains.  This  likely  resulted  in  an  underestimate  of  non- 
precipitation  clouds  in  the  summer,  hence  an  increase  in  solar  radiation  on  the  ground  and  an 
associated  increase  in  temperature  throughout  the  PBL. 

Although  the  overall  model  performance  is  better,  there  is  still  a  noticeable  cold  bias  and  lack  of 
diurnal  variability  in  the  model  results  for  the  winter  period.  The  primary  cause  is  likely  the  2.5 
degree  resolution  NCAR/NCEP  reanalysis  data  (NNRP).  Analyses  of  NCAR/NCEP  reanalysis 
data  (Kalnay  et  ai,  1996)  and  a  recent  model  evaluation  (Salathe,  2007)  suggest  that  there  is  a 
general  bias  towards  cold  temperatures  in  winter  in  this  dataset,  which  in  turn  can  create  biases 
towards  colder  temperatures  in  the  MM5  results. 

Domain  wide  average  temperatures  show  similar  patterns  (Appendix  B).  The  computed 
Fractional  Bias  (FB)  is  negative  in  winter  (indicating  a  general  under-prediction)  and  positive  in 
summer  (depicting  a  generalized  over  prediction).  The  net  effect  is  a  slightly  negative,  domain- 
wide  annual  average  FB. 

The  Normalized  Mean  Square  Error  (NMSE)  for  dry  bulb  temperature  is  higher  in  winter  than 
summer.  This  is  a  result  of  the  greater  magnitude  of  FB  for  this  period  and  also  likely  indicates  a 
greater  difference  in  station-observation  pairs. 
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4.0 


MODEL  RESULTS:  WIND  SPEED  AND  DIRECTION  -  SURFACE 


A  comparison  of  model  results  with  observations  is  complicated  by  the  fact  that  observations  of 
wind  direction  are  recorded  in  10's  of  degrees  and  wind  speeds  are  recorded  to  the  nearest  km/hr. 
Model  results,  on  the  other  hand,  are  available  to  the  nearest  tenth  of  a  unit. 

Paired  model  versus  observed  scatter  plots  for  both  wind  speed  and  direction  present  a  large 
degree  of  scatter  for  all  stations  (Appendix  A).  However,  the  time  series  plots  suggest  that  the 
model  predicted  the  majority  of  the  larger  scale  wind  flows  and  diurnal  patterns  relatively  well. 
The  one  exception  to  these  observations  is  for  Calgary  Airport. 

Scatter  plots  for  wind  direction  at  the  individual  stations  seem  to  show  better  agreement  in  the 
wintertime.  The  model  tends  to  over  predict  wind  speeds  in  winter,  with  the  exception  of 
Calgary  where  the  opposite  applies.  The  model  generally  under  predicted  wind  speeds  in  the 
summer  and  failed  to  capture  short  duration,  high  wind  speed  peaks. 

Predictions  from  stations  to  the  north  appear  to  show  larger  differences  between  the  predicted 
and  observed  than  stations  to  the  south  (with  the  exception  of  Calgary). 

Domain  wide  statistics  for  wind  speed  indicate  a  positive  FB  for  winter;  and  a  slight  negative  FB 
for  summer  (Appendix  B).  The  domain  wide  annual  average  FB  for  wind  speed  is  near  zero, 
suggesting  that  on  an  annual  basis,  wind  speed  is  generally  well  represented. 

The  NMSE  is  higher  for  wind  speed  in  the  summer  compared  to  the  winter,  pointing  to  a  higher 
degree  of  variation  between  model  results  and  observations  in  the  summer. 
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5.0 


MODEL  RESULTS:  RELATIVE  HUMIDITY  -  SURFACE 


The  model  significantly  over-predicts  relative  humidity  at  all  stations  during  the  winter.  This 
over-prediction  is  more  evident  for  the  more  northerly  stations.  Conversely,  the  model  under 
predicted  relative  humidity  in  the  summer.  Again,  the  predictions  are  worse  for  stations  further 
to  the  north. 

Time  series  for  summer  time  humidity  show  strong  diurnal  patterns  that  are  related  to  influences 
of  temperature  as  well  as  total  atmospheric  water  content.  These  same  patterns  are  not  evident 
for  the  winter,  which  is  to  be  expected. 

Domain-wide  time  series  show  that  predicted  RH  is  less  than  observed  even  when  predicted  and 
observed  temperatures  are  in  relative  agreement.  This  demonstrates  that  the  under-prediction  of 
RH  by  the  model  is  related  to  there  being  not  enough  water  vapour  in  the  model  atmosphere 
rather  than  strictly  a  temperature  dependence. 

Domain  wide  average  scatter  plots  show  almost  all  points  falling  above  the  1:1  reference  line  in 
the  winter,  indicating  a  strong  positive  bias  in  the  model  predictions.  Conversely,  most  points 
fall  well  below  the  1:1  reference  line  in  the  summer. 

Domain  wide  annual  average  FB  for  relative  humidity  is  nearly  zero,  which  indicates  that  the 
seasonal  over  and  under  predictions  tend  to  equal  out  over  the  course  of  the  year. 
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6.0 


MODEL  RESULTS:  UPPER  AIR 


A  comparison  of  model  results  to  upper  air  data  was  performed  based  on  two  stations:  Edmonton 
/  Stony  Plain,  Alberta;  and,  Great  Falls,  Montana.  Although  Great  Falls  is  not  in  Alberta  it  falls 
within  the  parent  MM5  domain  from  which  the  MMEU  meteorological  data  were  derived  and  as 
such  is  felt  to  be  relevant. 

The  evaluation  was  based  on  a  comparison  between  modelled  and  observed  values  of  wind 
speed,  wind  direction,  and  temperature  for  three  vertical  levels:  100m,  500m  and  1000m.  Unlike 
the  MM5  results,  which  are  available  hourly,  upper  air  observations  are  only  made  twice  daily,  at 
1200  and  0000  GMT  (Greenwich  Mean  Time).  In  addition,  observations  from  balloons  are  not 
recorded  at  pre-defined  vertical  levels;  rather,  observations  are  recorded  at  certain  pre-defined 
and  intermediate  pressure  levels  (not  specified  heights  above  ground).  Therefore,  the  exact 
height  from  the  MM5  model  outputs  (based  on  the  model's  vertical  grid  definition)  is  not  always 
available  in  the  upper  air  observations.  This  means  that  the  height  from  the  corresponding 
observation  measurements  typically  differ  by  10  to  100  meters  from  the  height  that  is  extracted 
from  the  model. 

Based  on  these  issues,  no  statistics  were  computed.  Instead,  time  series  graphs  depicting  model 
results  (pink  symbols  and  lines)  and  observations  (blue  diamonds)  were  used  to  perform  a 
qualitative  assessment  of  the  overall  model  performance.  Representative  graphs  for  summer 
(July)  and  winter  (January)  periods  are  provided  in  Appendix  C. 

Wind  directions  in  the  winter  are  replicated  well  at  all  levels.  Wind  directions  in  the  summer  are 
more  variable,  particularly  at  lower  levels.  This  is  because  meteorological  conditions  during  the 
winter  are  generally  driven  by  broad  synoptic  conditions  that  are  well  represented  by  MM5. 
Summertime  winds  at  lower  levels  are  more  dependent  on  boundary  layer  dynamics  associated 
with  stronger  thermal  affects  (surface  heating)  during  the  summer,  which  are  not  as  well 
represented  in  MM5.  This  causes  more  variability  between  model  results  and  observations 
during  the  summer  months.  Despite  the  greater  amount  of  'noise'  in  the  summertime  results,  the 
wind  patterns  are  captured  fairly  well  overall. 

Wind  speeds  are  well  represented  at  all  times  and  levels  with  the  model  reproducing  both  high 
and  low  wind  speed  patterns  with  no  apparent  biases.  The  model  also  appears  to  have  captured 
lower  overnight  winds  due  to  stable  night-time  conditions  in  summer  as  well  as  high  wind  speed 
events  due  to  frontal  passages  in  winter. 

Summertime  temperatures  and  diurnal  patterns  are  predicted  well  at  all  levels.  However,  aside 
from  a  short  period  in  early  January,  the  model  seems  to  underestimate  wintertime  temperatures 
at  all  levels.  This  could  be  associated  with  grid  resolution  issues  and  /  or  the  fact  that  the  upper 
air  observations  being  used  in  the  comparison  are  from  a  slightly  lower  elevation  than  the  model 
outputs,  as  discussed  above.  The  early  January  period  shows  a  marked  over-prediction  at  the 
100m  level  for  Edmonton  /  Stony  Plain,  presumably  associated  with  the  over  prediction  of  the 
intensity  of  a  warm  front  passage.  This  is  likely  the  result  of  the  model  bringing  more  warm  air 
to  ground  than  what  actually  occurred.  In  reality,  cold  surface  air  is  not  easily  dislodged  by  the 
passage  of  less  dense,  warmer  air  aloft.  Based  on  this,  the  timing  of  temperature  changes  at 
lower  levels  during  frontal  episodes  might  not  be  accurately  represented  by  the  model  (i.e.,  cold 
air  from  aloft  was  introduced  too  soon  in  the  case  described  above).  Similar  results  are  seen  in 
the  data  for  Great  Falls. 
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7.0  CONCLUSIONS 


In  general,  the  model  predictions  for  wind  speed,  direction,  and  temperature  are  adequate  both  at 
the  surface  and  aloft.  Temperatures  are  predicted  fairly  well  throughout  the  domain.  However 
on  a  seasonal  basis,  the  model  results  within  the  boundary  layer  are  generally  too  cool  and  moist 
during  the  winter  and  too  dry  and  warm  during  the  summer.  The  use  of  coarse  resolution  (2.5 
degree)  NCAR/NCEP  Reanalysis  Project  (NNRP)  data  as  the  primary  input  to  MM5  appears  to 
be  the  primary  cause  for  these  discrepancies.  These  problems  were  likely  exacerbated  by  the 
selection  of  the  MRF  PBL  scheme  and  by  omitting  the  shallow  convection  scheme  during  model 
execution.  Although  3-Dimensional  analysis  nudging  was  performed  for  the  parent  36  km 
resolution  domain,  no  surface  analysis  nudging  was  performed  for  either  the  36  km  or  12  km 
domains.  Theoretically,  FDDA  (Four  Dimensional  Data  Assimilation)  with  surface  nudging 
could  help  to  reduce  these  biases  in  the  12  km  results. 

Wind  speed  and  direction  depict  a  fair  degree  of  scatter  but  tend  to  align  well  with  the  observed 
trends  on  an  hour-by-hour  basis  (time  series),  both  at  the  surface  and  aloft.  Model  performance 
is  somewhat  better  in  the  southern  part  of  the  domain. 

Although  the  model  performance  for  relative  humidity  at  the  surface  appears  reasonable  on  an 
annual  average  basis,  it  is  in  fact  quite  poor  on  a  seasonal  basis.  The  under-predictions  in 
summer  and  over  predictions  in  winter  tend  to  cancel  out  in  the  annual  statistics. 

For  coarse  resolution,  annual  air  quality  modelling  scenarios,  Environment  Canada's  12  km 
resolution  MM5  output  for  2002  as  adopted  within  MMEU  should  be  adequate.  However, 
modellers  should  bear  in  mind  the  following  points  when  using  the  2002  MM5  dataset  for  air 
quality  or  meteorological  modelling.  This  holds  true  for  modellers  wanting  to  use  the  MM5 
outputs  either  through  the  MMEU  utility,  or  as  a  standalone  dataset. 

a)  Underestimations  of  wind  speed  in  summer  could  lead  to  reduced  turbulence  and  hence 
conservative  dispersion  predictions.  Conversely,  the  overestimation  of  wind  speeds  in 
winter  may  cause  air  quality  dispersion  models  to  predict  somewhat  on  the  low  side.  Wind 
speed  errors  are  generally  relatively  small,  however,  and  the  annual  bias  is  virtually  zero. 
Therefore,  although  there  are  some  variances  between  the  modelled  and  observed  on  an 
hourly  basis,  annual  dispersion  model  results  should  not  depict  any  particular  bias  due  to 
wind  speeds  from  MM5. 

b)  Errors  in  the  prediction  of  humidity  could  affect  secondary  chemistry  and  wet  deposition 
(depending  on  the  air  dispersion  model  and  physics  /  chemistry  algorithms  being  used). 
Warmer  and  drier  summer  predictions  may  potentially  increase  dispersion  and  reduce  wet 
deposition,  which  may  result  in  an  underestimate  of  short-term  maximum  ground  level 
pollutant  concentrations  and  /  or  total  deposition.  On  the  other  hand,  lower  predicted 
temperatures  in  the  winter  may  lead  to  more  stable  atmospheric  conditions,  causing  an 
overestimate  of  ground  level  concentrations  and  deposition  at  this  time  of  year.  Further, 
humidity  is  a  factor  in  the  growth  of  hygroscopic  (aqueous-phase)  secondary  particulate 
matter  and  therefore  (depending  on  the  air  quality  /  photochemical  model  being  used) 
affects  the  prediction  of  total  particulate  matter  and  calculations  pertaining  to  light 
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extinction  or  visual  range  (i.e.,  haze).  As  with  relative  humidity  itself,  this  issue  will  be 
more  prevalent  in  northern  parts  of  the  domain, 
c)     To  minimize  some  of  these  biases,  a  combination  of  MM5  with  available  surface 

observation  data  should  be  applied  in  air  quality  dispersion  modelling  applications.  With 
this  approach,  models  will  be  able  to  take  advantage  of  the  3-D  wind  fields  from  MM5, 
while  correcting  for  near  surface  wind  and  temperature  biases  using  local  observation  data. 
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